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(57) Abstract: A method and device that uses transform 
matrices to down- sample a DCT image direcdy in the DCT 
domain. The transfonn matrices have been selected to min- 
imize an optimization problem which is a function of the 
visual quality of down-sampled images obtained using the 
matrices and the computational complexity associated with 
using the transform matrices. The transfonn matrices com- 
prise a row transform matrix and a column transform ma- 
trix. A down-sampled image is produced by determining an 
intermediary matrix as the product of the DCT image and 
one of either the row transform matrix or the column trans- 
form matrix and then determining the down-sampled image 
as the product of the intermediary matrix and the transform 
matrix not already used to determine the intermediary ma- 
trix. 
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METHOD AND DEVICE FOR DOWN-SAMPLING A DCT IMAGE IN THE 

DCT DOMAIN 

FIELD OF THE INVENTION 

[0001] The present invention relates to image manipulation and 
more particularly to the down-sampling of an image to obtain an image 
having a smaller size. 

BACKGROUND OF THE INVENTION 

[0002] As wireless and wired network connectivity Is rapidly 
expanding and the number of network users Is steadily increasing, there 
has been great momentum in the multimedia Industry for supporting 
content display in a wide variety of network connected devices. This is 
complicated, however, due to the great diversity of devices with various 
display resolutions. The range of devices includes both standard and high- 
definition computer monitors with large displays, and smart phone devices 
with smaller displays. Accordingly, it is often necessary to down-sample 
an image to a size which may be displayed on a given device. Down- 
sampling may also be desirable in order to minimize the bandwidth or time 
required to transfer an image over a network. This is particularly a 
concern where the image is being transferred over a wireless medium. 

[0003] Many image and video types represent images or video in a 
Discrete Cosine Transform TDCT") domain. For example the 3PEG, MPEG, 
DV and M3PEG image and video types represent images or videos in the 
DCT domain. Where the image or video is represented in a DCT domain, 
down-sampling is particularly difficult. 

[0004] Typically, down-sampling a DCT image or video involves first 
converting DCT data to a spatial domain using an inverse DCT operation 
and applying a standard down-sampling method in the spatial domain. 
After the image is down-sampled in the spatial domain it is typically 
converted back to the DCT domain using a DCT operation. Due to the use 
of the DCT and the inverse DCT, methods which down-sample a DCT 
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image or video by first converting it bacl< to the spatial domain typically 
have a large computational complexity. A large computational complexity 
results in a longer calculation and a calculation which utilizes more 
processor time and power. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0005] Reference will now be made, by way of example, to the 
accompanying drawings which show an embodiment of the present 
application, and in which: 

[0006] Figure 1 shows a method of obtaining transforming matrices 
for down-sampling a DCT image according to one aspect of the present 
application; 

[0007] Figure 2 shows a method of obtaining a training set in 
accordance with one aspect of the present application; 

[0008] Figure 3 shows a method of selecting transform matrices in 
accordance with one aspect of the present application; 

[0009] Figures 4A and 4B show a three layer network representation 
of the transform matrices; 

[0010] Figure 5 shows a method of training transform matrices using 
a learning with forgetting algorithm; 

[0011] Figure 6 shows a method of training transform matrices using 
a selective learning with forgetting algorithm; 

[0012] Figure 7 shows a method of obtaining a down-sampled DCT 
image in accordance with one aspect of the present application; and 

[0013] Figure 8 shows an arrangement of electrical devices to be 
used in carrying out methods of the present application. 

[0014] Similar reference numerals are used in different figures to 
denote similar components. 
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DESCRIPTION OF SPECIFIC EMBODIMENTS 

[0015] In one embodiment of the present application, a method is 
provided for down-sampling a OCT image of size M x N directly in a DCT 
domain by applying two transform matrices. The first matrix, called a row 
transform matrix, Djm, is used to down-sample the number of rows in the 
DCT image. That is, it may be used to convert a DCT image, Cma/, with M 
rows into a DCT matrix with I rows. The number of columns in the row 
transform matrix is equal to the number of rows in the DCT Image. The 
second matrix, called a column transform matrix, Wi^, is used to down- 
sample the number of columns in the DCT image. That is, it may be used 
to convert a DCT image with N columns into a DCT matrix with J columns. 
The number of rows in the column transform matrix is equal to the 
number of columns in the DCT image. Accordingly, the row transform 
matrix is of a size I x M and the column transform matrix is of size N x J. 

[0016] The DCT image, Cmn, may be any DCT image including, for 
example, a JPEG image. In some instances the DCT image may represent 
a portion of a video, such as a frame of a video. For example, the DCT 
image may be a single frame of a video of the DV video type. 

[0017] A down-sampled DCT image is obtained using the transform 
matrices by multiplying the row transform matrix. Dim, with the product of 
the DCT image, Cmn, and the column transform matrix, W^j,. The down- 
sampled DCT Image may be obtained using a transformation formula: 

[0018] Since one of the transform matrices is applied to the DCT 
image on each side of the DCT image, the transform matrices may be 
called linear transform double sided matrices (LTDS). That is, one of the 
transform matrices is applied to the left of the DCT image in the 
transformation formula and one is applied to the right of the DCT image in 
the transformation formula. In this case, the row transform matrix is 
applied to the left of the DCT image and the column transform matrix is 
applied to the right of the DCT image. 

[0019] A wide range of spatial-domain down-sampling methods may 
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be represented as a LTDS in the DCT donnain. For example, in one 
embodiment, the transform matrices may be determined as: 

where Tss is a matrix of size S x S and which is equal to: 



^[/] 0 0' 
0 0 
.0 0 \i\ 



, where t is a DCT matrix. 



E/M and Gau are linear interpolation matrices for use in down-sampling in 
the spatial domain, and Amm is an M x M DFT transform matrix with its 

element given by a^^ = -—cxp^ "-^^^^ j,» = OX.„,M - 1, v = 0,1,...,M -1 and 

Alfj^ is its conjugate. Similarly, Bm is an N x N DFT transform matrix and 

Blff^ \s its conjugate matrix. Um and Rm are diagonal matrixes with 

diagonal elements being Lmi and Rin respectively. Uj and Rj/yare two one- 
dimensional filters. 

[0020] The LTDS representation above may be obtained by 
concatenating the inverse DCT, spatial domain down-sampling of a certain 
format, and the DCT. More specifically, a spatial domain image Xmn may 
be obtained by applying an inverse DCT to a DCT image Cmn- 
Xf^j^ ^fnCj^at, where t is a DCT matrix, t' is a transpose of the DCT 

matrix and □ denotes block-wise multiplications. Then, a spatial-domain 
method may be selected and used to down-sample Xmn to obtain an I x J 
image, denoted as Xjji Finally, the DCT may be applied to the I x J image, 
resulting in V^j =tax,jDt'., where Va is a down-sampled DCT image. 

[0021] Generally, down-sampling in the spatial domain consists of 
two steps: low-pass filtering and interpolation. In some embodiments, a 
low pass filter may be implemented based on a 2D discrete Fourier 

transform (DFT). Given an image Xmn, the filtering output Xm is obtained 

by: Xmn = A^^ ((^^^ ' ' ^m)® P'mn)' Kn * Where Amm is an M X M DFT 
transform matrix with its element given by 
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M 



,M = 0,l,...,M-l,v = 0,l,...,M-l 



and jC^^ is its conjugate. Similarly, Bm is an N x N DFT transform matrix 
and is Its conjugate matrix. F^yv is the low-pass filtering matrix in the 
DFT domain and the symbol ® denotes element wise multiplications. 

[0022] If ?MN is constructed using two one-dimensional filters (i.e. 
- ^M\ the element wise multiplication may be removed, 

yielding Xmn = A*^^ L^^ -{A^^ -X^^ B^^yR^^-B:^^, where and are 
diagonal matrixes with diagonal elements being Lmj and Rj/v respectively. 

[0023] A linear interpolation filter may then be applied. The linear 
interpolation filter to be applied is in the form of interpolation matrices, 
EiM, G/vj. Therefore, the spatial domain down-sampled image may be 
computed as 

~^\M '^MnGj^j - Af^f^ '^MM '^MN '^NN ' ^NN '^m ' ^ NJ 

[0024] The concatenation of inverse DCT, spatial domain down 
sampling and DCT is therefore: 

K =^D[£|A/ -^MA/ '^MM '{t'nC^^ut) 'B^^ 'R^N 'Bl,f, 'G^j]ar 

[0025] The block wise multiplication can then be replaced by 
applying a result of 



t ^Cm = '^MMyCMi^i where r^^^ = 



^[t] 0 0^ 
0 0 
0 0 [t]) 



and f is a DCT matrix. 



[0026] Consequently, a linear transform in the DCT domain can be 
obtained as: 

= D,^ ' C^, ^ W,j where D,^ = T,^ • E,^ - A^^ ■ L^^ • A^^ • T^^ , and 
[0027] That Is, the row transform matrix may be represented as: 
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Aa/ ^^iiy^iM '"^MM '^MM '^MM "^A/M./ - Similarly, the column transform 

matrix may be represented as W,, =7^^, ^B,, 'R,, 'B',, G,j -Tjj,,, The row 

transform matrix and the column transform matrix may be obtained by 
determining a solution to the equations: 

[0028] Accordingly, in one aspect of the present application a 
method of down-sampling a DCT image is provided. In one embodiment, 
the method includes steps of determining a row transform matrix using the 
equation: Z>,^ =T„^ E^^ -A*^^ ^L^^ -A^^ -T^^^, and determining a column 

transform matrix using the equation: = j;^, B^^ -R^^ B^^ • G^, -r,,,, 

[0029] As illustrated in Figure 1, in another embodiment of the 
present application, a method is provided for obtaining transform matrices 
for down-sampling a DCT image of size M x N to a down-sampled DCT 
image of size I x 3 directly in the DCT domain. 

[0030] The method of Figure 1 will permit a row transform matrix D/« 
and a column transform matrix to be selected which jointly maximize 
the visual quality of down-sampled DCT images that are obtained by 
applying the transform matrices to the DCT Images while minimizing the 
computational complexity of applying the transform matrices to the DCT 
Images to obtain the down-sampled DCT images. That is, a row transform 
matrix and a column transform matrix are selected which provide a desired 
trade-off between computational complexity and visual quality. It is 
typically considered desirable to have a higher quality down-sampled DCT 
image and a lower computational complexity associated with obtaining that 
down-sampled DCT image. 

[0031] At step 102 optimization parameters may be obtained. The 
optimization parameters permit a user, system administrator, or 
programmer to have control over the balance between computational 
complexity and visual quality. For example, the optimization parameters 
may include a computational complexity trade-off parameter, p, which is 
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used to balance the relative importance of various factors on the 
computational complexity. In some embodiments, the optimization 
parameters may include a rounding parameter, rj, which is used to 
determine how close a value must be to another value before the 
algorithm will attempt to round to the other value. The optimization 
parameters may also include a quality and complexity trade-off parameter 
A which is used to adjust the desired trade-off between the visual quality 
and complexity. That is, the quality and complexity trade-off parameter;i 
may be used to specify whether optimization of the transform matrices will 
be more or less focused on visual quality or complexity. The optimization 
parameters may also include selective learning with forgetting threshold 
parameters, w©, do, , which will be used in a selective learning with 
forgetting stage to adjust the threshold above which an element in the 
transform matrices will be considered protected and not pushed towards 
zero. In some embodiments, the optimization parameters may include a 
learning step size parameter, a , which is used to adjust the step size of 
learning operations. That is, it is used to adjust how great the degree with 
which the transform matrices will be adjusted after an iteration of learning. 

[0032] Depending on the method used to optimize the transform 
matrices, some or all of the optimization parameters discussed above may 
be obtained in step 102. In some embodiments, these parameters are 
stored in a memory and they are retrieved from the memory. 

[0033] In other embodiments, the optimization parameters may be 
obtained from a user. A user may be prompted to enter desired 
optimization parameters using an input device. In some embodiments, a 
setup rnodule may be run by the user which allows the user to enter 
optimization parameters. After the user enters the optimization 
parameters, they may be saved to the memory. 

[0034] In some embodiments, the optimization parameters are 
entered on a server which obtains the transform matrices. 

[0035] Next, at step 104, a training set is obtained which will be 
used in training the transform matrices. The training set includes at least 
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one DCT reference image, Cmn, and typically includes a corresponding 
down-sampled DCT reference image, Vu, for each reference image. The 
DCT reference Image is of the size MxN and the down-sampled DCT 
reference image is of the size IxJ. 

[0036] As will be described more fully below, the training set will be 
used to evaluate the quality of down-sampled images obtained using the 
transform matrices relative to the quality of the down-sampled DCT 
reference image. 

[0037] Typically, the training set will consist of more than one DCT 
reference image Cmn and more than one down-sampled DCT reference 
image, Vu. The down-sampled DCT reference image Vu is obtained using 
another down-sampling method. Typically, the down-sampled DCT 
reference image Vu is obtained using a spatial domain reference image in 
which the DCT reference image Cmn is converted to the spatial domain 
using an inverse DCT transform, and the spatial domain image is down- 
sampled using a spatial domain down-sampling method. Then, the down- 
sampled spatial domain reference image is transferred back to the DCT 
domain using a DCT transform to obtain the down-sampled DCT reference 
image, Vu. 

[0038] This will permit the transform matrices to be trained multiple 
times, thus increasing the accuracy of the transform matrices. For 
example, in some embodiments the training set includes five DCT 
reference images Cmn and five down sampled DCT reference images, Vu, 
each corresponding to one of the DCT reference images. 

[0039] The step 104 of obtaining a training set may include a step of 
applying a pre-selected down sampling method to a DCT reference image 
to obtain the down-sampled DCT reference images. Typically, the pre- 
selected down-sampling method is a spatial domain down-sampling 
method. 

[0040] Referring now to Figure 2, a method is shown for obtaining a 
training set using a spatial domain down-sampling method. At step 202 a 
down-sampling method is selected. Typically, the selected down-sampling 
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method will be a down-sampling method which produces a down-sampled 
OCT reference image with an optimal visual quality since the visual quality 
of the down-sampled image produced according to the present method is 
dependent on the visual quality of the down-sampled DCT reference 
image. 

[0041] In some Instances, the down-sampling method is pre- 
determined. 

[0042] Where the down-sampling method is a spatial domain down- 
sampling method, the method of obtaining the training set includes a step 
204 of converting at least one of the DCT reference images to the spatial 
domain by applying an inverse DCT to produce a spatial domain reference 
image. Next, at step 206, the spatial domain reference image is down- 
sampled using the selected spatial domain down-sampling method to 
produce a spatial domain down-sampled reference image. Then, at step 
208, the spatial domain down-sampled reference image Is converted back 
to the DCT domain to obtain the down-sampled DCT reference image. 

[0043] Typically, the step 206 of down-sampling the converted DCT 
reference image in the spatial domain includes the steps of: 1) applying a 
low pass filter to the spatial domain reference image to produce a filtered 
image; and 2) interpolating within the filtered image to produce the spatial 
domain down-sampled reference Image. 

[0044] With respect to the step of filtering, a filter is typically 
selected which provides a good trade-off among three factors: the 
reduction or elimination of aliasing, the preservation of low-frequency 
components, and the reduction or elimination of ringing. Generally, a filter 
with a sharp transition band provides a good performance on anti-aliasing 
and preserving low-frequency components, yet a sharp transition band 
incurs ringing along intensity edges in the filtered image. 

[0045] In some embodiments, a Butterworth filter is used. The 
Butterworth filter provides a good trade-off among the three factors 
(aliasing, low-frequency components, and ringing). That is, two one 
dimensional Butterworth filters may be selected with the frequency 
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response function: 

where is the cutoff frequency and L represents the order of the filter 

which characterizes the transition band. In some embodiments, the filter 
may be a tenth order filter (i.e. L=10). 

[0046] As noted above, this filter may be applied to the spatial 
domain reference image to obtain a filtered image. 

[0047] Following the step of low-pass filtering, interpolation may be 
performed on the filtered image to obtain the spatial domain down- 
sampled reference image. Interpolation involves estimating the values of 
the spatial domain down-sampled reference image at certain points. 

[0048] A wide range of interpolation methods may be used. For 
example, nearest neighbour interpolation or spline interpolation may be 
used. The Nyquist-Shannon interpolation uses a sine function which 
achieves theoretically optimal performance. 

[0049] In some embodiments, cubic interpolation may be used which 
provides a smooth function with a continuous first derivative. While the 
nearest neighbor interpolation corresponds to a square interpolation 
function, the bicubic interpolation is obtained by convolving three square 
functions. 

[0050] It will be apparent to a person skilled in the art that, while 
specific spatial domain down-sampling methods have been discussed 
above, the method discussed herein may be used with any other spatial 
domain method. 

[0051] The training set may be stored in a memory after it is 
created. In some embodiments, the memory may contain a database of 
training sets for various down-sampling ratio. For example, the database 
may contain data training sets to train transform matrices to convert a 
matrix of size M x N to a matrix of any one of a number of various sizes 
(i.e. I X J, O X P, Q X R, etc.). 
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[0052] In embodiments where training sets are previously generated 
and stored in memory,, tlie step 104 of obtaining a training set may merely 
require the training set to be retrieved from the memory. 

[0053] Once a training set is obtained, the method of obtaining 
transform matrices for down-sampling a DCT image Includes a step 106 of 
selecting transform matrices that minimize an optimization problem. The 
optimization problem is a function of the error between the down-sampled 
image obtained by applying the transform matrices to the reference image 
and the down-sampled DCT reference image. The optimization problem is 
also a function of the computational complexity of applying the transform 
matrices. 

[0054] It will be appreciated that various methods may be employed 
to find transform matrices which jointly optimize the visual quality and 
computational complexity. In some embodiments, discussed more fully 
below, a multi-layer network may be used to find transform matrices which 
solve the optimization problem. 

[0055] Accordingly, in step 106, transform matrices are found which 
obtain a suitable balance between maximizing the quality of the down- 
sampled image obtained using the matrices, while minimizing the 
computational complexity associated with down-sampling using the 
transform matrices. That is, an LTDS is found with the best trade-off 
between the fidelity of a down-sampled image and the computational 
complexity of the using the LTDS in the sense of minimizing the joint cost. 

[0056] In order to select the transform matrices which minimize the 
joint cost, a value representing the visual quality of the down-sampled DCT 
images obtained using the transform matrices may be determined. A 
value representing the computational complexity associated with applying 
the transform matrices to a DCT image in order to down-sample the DCT 
image may also be determined. The joint cost of the transform matrices 
may then be determined as a function of the value representing the visual 
quality of the down-sampled DCT images obtained using the transform 
matrices and the computational complexity associated with applying the 
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transform matrices to a DCT image in order to down-sample the DCT 
image. 

[0057] The visual quality of the down-sampled image is measured 
relative to the quality of the down-sampled DCT reference image obtained 
using the other down-sampling method. Where quality is measured in this 
manner, the linear transform double sided matrices, Dim and W^j, may be 
selected to minimize the difference in between the quality of the image 
obtained using the transform matrices (i.e. D;^ C^^ -^vy) and the quality 
of a down-sampled DCT reference image, Vu, obtained using the other 
down-sampling method. That is, the transform matrices. Dim and Wnj, are 
selected to satisfy the following optimization problem: 



where -Cf^^f -Wj^ is a down-sampled image obtained using the 
transform matrices, V,j is the DCT reference image which represents a 
down-sampled image obtained using another down-sampling method, 

is a measure of the computational complexity associated with using the 
transform matrices Df^^.W^j to down-sample an image Cmn and, A, is a 
quality and complexity trade-off parameter. 

[0058] The quality and complexity trade-off parameter, A, may be 
used to balance the trade-off between the visual quality and computational 
complexity. That is, the quality complexity trade-off parameter may be 
used to adjust whether optimization will be more or less focused on visual 
quality or complexity. 

[0059] As used in the above equation, the quality and complexity 
trade-off parameter, A, is applied to the computational complexity 
associated with using the transform matrices. In this case, a greater 
quality complexity trade-off parameter, A, will result in more importance 
being placed on complexity. Where the quality complexity trade-off 
parameter, X, is applied in this manner, an example of a suitable value 
may be 0.1. However, it will be appreciated that various values for the 
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quality and complexity trade-off parameter, A, may also be used. 

[0060] Where a quality and complexity trade-off parameter. A, Is 
used, the method may further include a step of retrieving the quality and 
complexity trade-off parameter. This step may be included in the step 102 
of obtaining the optimization parameters. The quality complexity trade-off 
parameter, may be retrieved from a memory. In other embodiments, 
the a quality and complexity trade-off parameter, A, may be obtained via 
user input. 

[0061] The computational complexity may be a function of the 
absolute values of elements in the row-size transform matrix and the 
column transform matrix. That is, in some embodiments, the 
computational complexity may be calculated as: = j^/w | + |^a/j |# where 
I I defines the U norm of a matrix. 

[0062] The computational complexity may, therefore, be a function 
of how far the elements of the matrices are from zero. It will be 
appreciated that the computational complexity of the transform matrices 
will be minimized when all elements of the transform matrices are zero. Of 
course, it is undesirable to have transform matrices where all elements are 
zero since it would not produce a down-sampled image and the quality, 
would therefore, be undesirable. 

[0063] To determine how far the elements of the matrices are from 
zero, the method may include a step of calculating the summation of the 
absolute values of elements in the transform matrices. 

[0064] In some embodiments, the complexity model Is established to 
ignore the non-zero effect of matrix elements with values greater than a 
threshold Wo, do on computational complexity. For example, the 
complexity model may be: 

[0065] That is, the method may comprise a step of calculating the 
summation of the absolute value of all elements in the rdw transform 
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matrix and the column size nnatrix that are below two thresholds Wq, do. 
In many cases, the thresholds may be the same (I.e. Wo= do). 

[0066] Besides the number of non-zero elements in the row 
transform matrix and the column transform matrix, the complexity for 
computing a down-sampled DCT image using the transform matrices is 
also related to how multiplications may be implemented. In general, a 
multiplication may be approximated by a series of additions and shifts. 

[0067] In some embodiments, the computational complexity may be 
related to the number of non-zero elements in the binary representation of 
the number. That is. It is considered more desirable to have a zero in a 
binary representation than a one. Accordingly, the computational 
complexity may evaluate the number of non-zero elements in the binary 
representation of each of the row transform matrix and the column 
transform matrix. This may be done by locating a computationally more 
desirable value within the neighborhood of a current element of the 
transform matrix. The computational complexity of the value may be a 
function of the difference between the current element of the transform 
matrix and the computationally more desirable value. 

[0068] More particulariy, for any element, b, in one of the transform 
matrices, a set of binary digits, {a,}, may be found which satisfies the 
equation: 

k} = arg min ZKI 

where n is a rounding parameter, which is used to adjust how large the 
neighborhood of an element, b, may be. That is, n is used to adjust how 
close the element, b, must be to another value having more zeros in its 
binary representation before the other value will be considered desirable. 
The binary representation may be converted back into a decimal number 
system representation to obtain a quantized decimal value. The procedure 
of locating a neighboring value which is less complex than the current 
element, b, may be referred to as a quantization procedure, Q(b). 

[0069] In some embodiments, a binary number system is used in 
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which every binary digit may be signed. That is, the binary digits may be 
an element of the set consisting of positive one (1), zero (0), or minus one 
(-1). (i.e. a. e (1,-1,0)). In some embodiments, such a binary 

representation may be used since it permits binary representations with a 
maximum number of zeros (0). For example, if we consider the binary 
representation of the number 0,46875 in a binary system in which each 
element must be selected from the set consisting of only one (1) and zero 
(0), the binary representation would be 0.01111. That Is, 

0.46875 = (0 X 2°) + (0 x 2"^) + (1 x 2"^) + (1 x T^) + (1 x 2"*) + (1 x 2'^) 

Using this number system, four ones are needed to represent the number. 

In contrast, the same number could be represented using the signed 
system mentioned above using only two ones: 

0.46875=(0 X 2*^)+(l x 2-^)+(0 x r2)+{0 x 2 3)+(0 x 2-^)+((-l) x 2'^). 

[0070] Typically, the magnitude of all elements in the transform 
matrices Djj^,W^j is in the range [0, 8). That is, the magnitudes of all 

elements is greater than or equal to 0 and less than 8. If this assumption 
is made, then the integer portion of the elements may be represented with 
three binary digits (i.e. 2^ 2\ 2°). The number of binary digits used to 
represent the decimal fraction portion of the elements may depend on the 
desired visual quality of down-sampled DCT images obtained using the 
matrices. For example, if fewer binary digits are used to represent the 
decimal fraction portion of the elements, the computational complexity 
associated with using the transform matrices will be reduced, but the 
visual quality of the down-sampled DCT images may also be reduced. In 
some embodiments, fifteen (15) binary digits are allocated for 
representing the decimal fraction portion of the elements of the transform 
matrices. 

[0071] If the binary system discussed above is used, the 
quantization procedure for an element, b, may be represented as: 



wo 2008/148205 



PCT/CA2008/001080 



- 16 - 

= • 2-\ a, G 0,-1,0), where {a,}=:arg mill EKI 

[0072] That is, the quantization procedure referred to above 
determines a set of binary numbers, {ai} for a computationally more 
desirable value. Then the decimal representation of the computationally 
more desirable value is determined. 

[0073] The difference between the current element, of the 
transform matrix and the computationally more desirable value may then 
be found. For example, the computational complexity associated with the 
number of non-zero elements in the binary representation of the elements 
of the transform matrices may be determined as: 

where Q(Dim), Q(Wn3) mean to apply a quantization procedure to each 
element of the row transform matrix and the column transform matrix 
respectively. 

[0074] Since the computational complexity is a function of both the 
number of non-zero decimal representations in the transform matrices and 
the number of non-zero elements in the binary representation of the 
elements of the transform matrices, the computational complexity may 
determined as: 

where p is a computational complexity tradeoff parameter used to balance 

the relative importance on the computational complexity of the number of 
non-zero elements in the transform matrices and the difference between 
current weights in the transform matrix and weights that are close to the 
current weights but which would have more zeros in their binary 
representation. In some embodiments, p is equal to 0.5. 

[0075] Referring now to Figure 3, a method of obtaining transform 
matrices according to one aspect of the present application is shown. In 
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this embodiment, a multiple-layer network is used to find transform 
matrices which minimize the joint cost of the transform matrices. That is, 
a multiple-layer network is used to find transform matrices which offer a 
desired balance between computational complexity and visual quality. 

[0076] At step 302, transform matrices are initialized. Referring to 
Figures 4A and 4B a multiple-layer network representation of the 
transform matrices is shown. A three layer network structure is used, in 
which there is an input layer 402, representing an input image Cmn of size 
1^ X N. There is also a hidden layer 404, representing an intermediary 
matrix Yin of size I x N. The multiple-layer network structure also contains 
an output layer 406, representing a down-sampled Image Zu of size I x J. 
Connections 408, 410 are selectively built up among units in each two 
layers to simulate the matrix multiplication operation in the linear 
transform. That is, connections 408 between the input layer 402 and the 
hidden layer 404 represent the application of the row transform matrix Dim 
tp the input image Cmn- Similarly, connections 410 between the hidden 
layer 404 and the output layer 406 represent the application of the column 
transform maitrix Wnj to the intermediary matrix Yin. 

[0077] Connections between the layers are established according to 
a number of rules. With respect to the connections 408 between the input 
layer and the hidden layer, connections are established from units in a 
given column (i.e. nj) of the input layer to units of the same column (I.e. 
ni) in the hidden layer. That is, the input layer and the hidden layer have 
the same number of columns. Units in a given column of the input layer 
are fully connected to units in the same column of the hidden layer. Valid 
connections between any of two columns of the input layer and the hidden 
layer share the same weight matrix i.e. Dim. 

[0078] Similarly, connections 410 between the hidden layer 404 and 
the output layer 406 are built up among rows. Connections are 
established from units in a given column (i.e. iO of the hidden layer 404 to 
units in the same column of the output layer 406 (i.e. ii). Valid 
connections 410 between any two rows of the hidden layer and the output 
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layer share the same weight matrix i.e. Wnj. 

[0079] Accordingly, the output of the hidden layer 404 is computed 
as Yifi = D^f^ • Cf^fj and the output of the output layer 406 is computed as 

[0080] Referring again to Figure 3, during the step of initialization, 
an initial starting weight may be placed In the elements of the transform 
matrices Dim, Wn],. The initial starting weight represents a starting point 
from which a decay procedure will be applied. In some embodiments, 
random weights may be placed In the elements of the transform matrices 
Dim, Wnj. 

[0081] At step 304, a learning with forgetting algonthnr) Is applied to 
the transform matrices to obtain transform matrices with a reduced 
learning objective function. This stage is used to find a skeleton structure 
of the transform matrices. 

[0082] It will be appreciated that due to a random initialization of the 
connection weights, some redundant connections may possess an Initial 
weight with a large value. Accordingly, the learning with forgetting stage 
will apply a constant decay to all elements In order to remove redundant 
connections which are present due to a random initialization of the 
weights. Such redundant connections are removed as much as possible. 
This approach leads to a constant decay for all non-zero elements, forcing 
as many elements to be zero as possible. 

[0083] The learning objective function at this stage is: 

where Jj- is the learning objective function, D^^^, is the row transform 
matrix, W^^j is the column transform matrix, C^^ is the DCT image, 
Aw '^m'^Nj "5 the down-sampled DCT image obtained using the 
transform matrices, V^j is a down-sampled DCT reference image obtained 
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using another down-sampling nriethod, is the computational complexity 
associated with using the transform matrices D,j^,Wf^j to down-sample an 

image Cmn and A is a quality and complexity trade-off parameter used to 
balaince a trade-off between visual quality of down-sampled DCT images 
and the computational complexity of obtaining the down-sampled DCT 
images. In this case, the computational complexity is a function of the 
absolute value of all elements of the transform matrices. That is, 
Kg =|^/wl + |^AL/|/ where | | defines the /i. norm of a matrix. 

[0084] Since the complexity for this stage is a function of the 
absolute value of all elements in the matrices, the complexity is minimized 
if every element is zero. 

[0085] It will also be noted that the visual quality at this stage is 
measured as the mean square error of the visual quality of the down- 
sampled DCT image produced using the transform matrices and the visual 
quality of the down-sampled DCT image produced by applying another 
down-sampling method to the same initial image. 

[0086] As illustrated in Figure 5, the learning with forgetting 
algorithm includes a step of adjusting matrices by a learning amount. The 
learning amount may be calculated by using a back propagation algorithm. 

[0087] First, the input signal is passed forward to compute network 
outputs: 

[0088] A network error is then determined by comparing the network 
output to the output for the down-sampled DCT reference image, which 
was produced using another down-sampling domain method. The network 
error is then propagated backward. 

AZ,, =Z,j -V,j =>(Ay),^ =(AZ),, -{W^),, 

[0089] The learning amounts may then be determined for each of 
the transform matrices. The learning amounts may be determined as: 
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where sgn(x) is the sign function as: 



-,a:>0 
2 

0,;c = 0 

--,x<0 
2 



[0090] The learning amounts may then be applied to the transform 
matrices to obtain adjusted transform matrices: 



where the superscripts (n) and (n+1) accord to the nth and (n+l)th 
iterations and a is a small positive number named the learning factor or 
learning step size parameter. The learning factor is used to adjust the 
step size of learning operations. That is, it is used to adjust how great the 
adjustment will be with each iteration of learning. In some embodiments, 
the learning factor is 1x10"^. 

[0091] Typically, there will be multiple learning with forgetting 
iterations. In one embodiment, at step 504, a determination is made 
regarding whether the decrement of the learning objective function was 
less than a predetermined threshold. If it was not, another iteration of 
learning with forgetting will be performed. If it was less than the 
predetermined threshold, then the learning with forgetting stage will be 
considered to be completed. That is, the learning with forgetting stage is 
complete when each iteration of learning no longer yields a sufficient 
improvement to the transform matrices in terms of minimizing the joint 
cost. 
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[0092] In some embodiments, the predetermined threshold is zero. 
That is, the learning objective function will be completed when the joint 
cost of the transform matrices is no longer improving with each iteration. 

[0093] Learning with forgetting normally ends with a skeleton 

structure but a large distortion of \\D-C'W-vf , That Is, a constant decay 

to elements with large values will introduce a large distortion to the visual 

quality, measured as l^,j -T'\Dfj^^ -Cj^^ •^^J)f / where Xy is a spatial 

domain down-sampled image and T^(x) is an inverse DCT transform. 

[0094] As illustrated in Figure 3, a step 306 of performing a selective 
learning with forgetting algorithm may then be used to tune the structure 

for better trade-off between the distortion of \D'C'W -vf^r\6 the 

complexity. The selective learning with forgetting stage protects certain 
large elements from decay so that they can be trained to focus on 
providing better visual quality. 

[0095] Since the selective learning with forgetting stage protects 
certain large elements from decay, the complexity model may include at 
least one selective learning with forgetting threshold parameter (Wo or do) 
to be used for determining what is or is not considered a large element. In 
some embodiments, each of the transform matrices has its own selective 
learning with forgetting threshold parameter (Wo or do). In other 
embodiments, the selective learning with forgetting threshold parameter 
(Wo or do) is the same for both transform matrices. 

[0096] The selective learning with forgetting threshold parameter (Wq 
or do) has the effect of adjusting the trade-off between visual quality and 
complexity since a higher threshold may result in a higher quality image, 
but will result in less elements of the transform matrices which have been 
pushed toward zero. In some embodiments, the selective learning with 
forgetting threshold parameter (Wo or do) is 0.1. 

[0097] The learning objective function at this stage is very similar to 
the learning objective at the learning with forgetting stage: 
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where J ^ is the learning objective function, is the row transform 

matrix, is the column transform matrix, C^j^ is the DCT image, 

^m '^m'^m the down-sampled DCT image obtained using the 

transform matrices, V^j Is a down-sampled DCT reference Image obtained 

using another down-sampling method, Is selective learning with 

forgetting computational complexity associated with using the transform 
matrices D^^^.W^j to down-sample an Image Cmn and A is a quality and 

complexity trade-off parameter used to balance a trade-off between visual 
quality of down-sampled DCT images and the computational complexity of 
obtaining the down-sampled DCT images. 

[0098] In this case, however, the complexity model is established to 
ignore the non-zero effect of matrix elements with values greater than the 
threshold on the complexity. For example, the complexity model for the 
learning with forgetting stage may be: 



[0099] That is, the method may comprise a step of calculating the 
summation of the absolute value of all elements in the row transform 
matrix and the column size matrix that are below two thresholds Wo, do. 
In many cases, the thresholds may be the same (i.e. Wo= do). 

[00100] As discussed previously, besides the number of non-zero 
elements in the row transform matrix and the column transform matrix, 
the complexity for computing a down-sampled DCT Image using the 
transform matrices is also related to how multiplications may be 
implemented. The computational complexity may be related to the 
number of non-zero elements in the binary representation of the number. 
That is, it is considered more desirable to have a zero in a binary 
representation than a one. Accordingly, the computational complexity may 
evaluate the number of non-zero elements in the binary representation of 
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each of the row transform matrix and the column transform matrix using 
the algorithm discussed above. This may be done by locating a 
computationally more desirable value within the neighborhood of a current 
element of the transform matrix. The computational complexity of the 
value may be a function of the difference between the current element of 
the transform matrix and the computationally more desirable value. 

[00101] More particularly, for any element, b, in one of the transform 
matrices, a set of binary digits, {a/}, may be found which satisfies the 
equation: 



where n is a rounding parameter, which is used to adjust how large the 
neighborhood of an element, b, may be. That is, n is used to adjust how 
close the element, b, must be to another value having more zeros in its 
binary representation before the other value will be considered desirable. 
The binary representation may be converted back into a decimal number 
system representation to obtain a quantized decimal value. The procedure 
of locating a neighboring value which is less complex than the current 
element, ft, may be referred to as a quantization procedure, Q{b). 

[00102] If the binary system discussed above is used, the 
quantization procedure for an element, b, may be represented as: 



[00103] That is, the quantization procedure referred to above 
determines a set of binary numbers, {aj for a computationally more 
desirable value that is within the neighborhood of the element, b. Then 
the decimal representation of the computationally more desirable value is 
determined. 

[00104] The difference between the current element, 6, of the 
transform matrix and the computationally more desirable value may then 
be found. For example, the computational complexity associated with the 




Q(b) = Xa, . 2-' ,a, e {1,-1,0), where {a,} = arg niin EkI 



\b-Y,ia,2'')\^b\n 
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number of non-zero elements in the binary representation of the elements 
of the transform matrices may be determined as: 

where Q(Dim), Q(Wnj) mean to apply a quantization procedure to each 
element of the row transform matrix and the column transform matrix 
respectively. 

[00105] Since the computational complexity is a function of both the 
number of non-zero decimal representations in the transform matrices and 
the number of non-zero elements in the binary representation of the 
elements of the transform matrices, the computational complexity in the 
selective learning with forgetting algorithm may determined as: 

where /? is a computational complexity tradeoff parameter used to balance 
the relative importance on the computational complexity of the number of 
non-zero elements in the transform matrices and the difference between 
current weights in the transform matrix and weights that are close to the 
current weights but which would have more zeros In their binary 
representation. In some embodiments, p is equal to 0.5. 

[00106] The quantization procedure discussed above, which examines 
whether there is a neighboring value which is computationally less 
complex, is generally applied at the selective learning with forgetting stage 
because its corresponding contribution to the complexity function is at a 
level similar to =|i)/^,|,,j,,, +I^a^J|..|<.. which is much less than 

[00107] Note, however, that regardless of which stage the 
quantization procedure is applied at, the computational complexity tradeoff 
parameter p may be used to balance the relative importance on the 
computational complexity of the number of non-zero elements in the 
transform matrices and the difference between current weights in the 
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transform matrix and weights that are close to the current weights but 
which would have more zeros in their binary representation. 

[00108] Referring now to Figure 6, a method of applying a selective 
learning with forgetting algorithm. to the transform matrices is shown. At 
step 602, the transform matrices are adjusted by a learning amount. 

[00109] The learning amount is determined in much the same way as 
the learning amount is determined for the learning with forgetting stage. 

[00110] First, the input signal is passed forward to compute networl< 
outputs: 

[00111] A network error is then determined by comparing the network 
output to the output for the down- sampled DCT reference image, which 
was produced using another down-sampling domain method. The network 
error is then propagated backward. 

[00112] The learning amounts may then be determined for each of 
the transform matrices. The formula for determining the learning amounts 
in the selective learning with forgetting stage differs from the formula for 
determining the learning amounts for the learning with forgetting stage. 
In the selective learning with forgetting stage, the learning amounts may 
be determined as: 

AD = i^Y),, • (C + ;i . thr{D,^ .dJ^^ 'P- sgnCA^ " )) 



where: 



thr{x,d) = 



^,e>x>0 
2 

0,x^Q;x^e;x<-9 

-~,-^<a:<0 
2 
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sgn{A:) = ^ 



-,x>0 

0,x = 0 




and A is the quality and complexity trade-off parameter and^ is the 
computational complexity tradeoff parameter. 

[00113] As before with the learning with forgetting algorithm, the 
learning amounts may then be applied to the transform matrices to obtain 
adjusted transform matrices: 



where the superscripts (n) and (n+1) accord to the nth and (n-f l)th 
iterations and a is a small positive number named the learning factor or 
learning step size parameter. As before, the learning factor is used to 
adjust the step size of learning operations. The learning factor for the 
selective learning with forgetting stage may be a different value than the 
learning factor for the learning with forgetting stage. 

[00114] Typically, there will be multiple selective learning with 
forgetting iterations. In one embodiment, at step 604, a determination is 
made regarding whether the decrement of the learning objective function 
was less than a predetermined threshold. If it was not, another iteration 
of learning with forgetting will be performed. If it was less than the 
predetermined threshold, then the selective learning with forgetting stage 
will be considered to be completed. That is, the selective learning with 
forgetting stage is complete when each iteration of learning no longer 
yields a sufficient improvement to the transform matrices in terms of 
minimizing the joint cost. 

[00115] In some embodiments, the predetermined threshold is zero. 
That is, the learning objective function will be completed when the joint 
cost of the transform matrices is no longer improving with each iteration. 
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[00116] Following the selective learning with forgetting algorithm. In 
some embodiments the computational complexity of the transform 
matrices may be further reduced by combining arithmetic operations. 

[00117] Referring now to Figure 7, according to one aspect of this 
application, a method is provided for down-sampling a DCT image of size 
X N in the DCT domain to produce a down-sampled image of size I x J. 

[00118] In step 702, transform matrices are obtained. The transform 
matrices Include a row transform matrix of size I x M and a column 
transform matrix of size N x 3. The transform matrices may have been 
produced according to any of the methods of producing transform matrices 
for down-sampling a DCT image directly in the DCT domain outlined 
above. That is, the row transform matrix and the column transform matrix 
are matrices that have been selected to jointly optimize the visual quality 
of down-sampled DCT images obtained using the matrices and minimize 
the computational complexity associated with applying the matrices to DCT 
images. The visual quality of the down-sampled Images is determined 
with reference to the visual quality of another down-sampled image 
obtained using another down-sampling method. In some embodiments, 
the matrices have been determined using a multiple-layer network to solve 
an optimization problem. 

[00119] In some embodiments, the step 702 of obtaining includes 
looking up the row transform matrix and the column transform matrix in a 
memory. In other embodiments, the step 702 of obtaining includes 
looking up appropriate transform matrices in a database of transform 
matrices. In such embodiments, the database may include transform 
matrices for various down-sampling ratios and the step of obtaining 
includes selecting the transform matrices with a desired down-sampling 
characteristic, such as a desired down-sampling ratio. 

[00120] In some embodiments, the database may be stored remotely 
on a server. The step of obtaining includes a step of requesting 
appropriate transform matrices from the server and receiving the row 
transform matrix and colunin transform matrix from the server. To ensure 
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that the correct transform matrices are obtained, the step of requesting 
may include transmitting system identification data to the server. For 
example, the identification data may include data identifying a display 
resolution of a device which will be used to display the down-sampled 
image and/or data identifying processor speed information of a processor 
which will be used to transform the DCT image into the down-sampled DCT 
image. 

[00121] In this way, transform matrices may be received which are 
best suited for the device's display or processing capabilities. 

[00122] Following the step of obtaining, a step 704 of determining an 
intermediary matrix as the product of the DCT image and one of either the 
row transform matrix or the column transform matrix. Since matrix 
multiplication is not commutative, the arrangement of multiplication may 
be important. The row-transform matrix, Dim, is generally to the left of the 
DCT image, Cmn, in the transformation formula and the column-transform 
image, Wni is generally to the right of the DCT image in the 
transformation formula. 

[00123] That is, since£>,^ C^^ ^ ^^j C^^ D,^ , the latter 
expression Is not used. However, since matrix multiplication is associative, 
the order may be modified to some extent. The row size matrix may be 
multiplied with the DCT image and the result multiplied with the column 
transform matrix (I.e. (D^^ •C^Ar)^^/ )/ or alternatively the DCT image may 
be multiplied with the column transform matrix and the result multiplied 
with the row transform matrix (i.e. D;^ -(C^^ • Wj^)). 

[00124] Accordingly, in step 704 an intermediary matrix, K, is 
obtained as the product of the DCT image and one of either the row 
transform matrix or the column transform matrix is obtained as either: 

[00125] Next, at step 706, the down-sampled DCT image is obtained 
as the product of the intermediary matrix K and the transform matrix that 
is not used in this previous step of determining (either D/m or Wnj). For 
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example, if the intermediary matrix K is determined as K= (D^^ • Cj^^^), 
then the down-sampled OCT image is obtained as Xu= {K -W^j). 
Alternatively, if the intermediary matrix K is determined as K= iCn^j^ lV^), 
then the down-sanipled DCT image is obtained as Xu=(D/^ K) . 

[00126] It will be appreciated that transcoders for down-sampling 
images are used in wide variety of electrical devices. Accordingly, the 
methods described above may be implemented on any suitable electronic 
device comprised of a processor and a memory. 

[00127] By way of example, in one embodiment, illustrated in FIG. 8, 
the methods described above are used to down-sample images to a size 
suitable for display on a display of a mobile device 802. The mobile device 
802 is connected to a server 804. The mobile device 802 may be 
connected to the server 804 in any one of a number of methods. For 
example, the mobile device 802 may be connected to the server through a 
cellular network, a wireless wide area network (WWAN), or a wireless local 
area network (WLAN). It will be appreciated that other methods of 
connection, both wired and wireless, may also be used. 

[00128] The server 804 may be an Internet gateway which provides 
access to the Internet. In some embodiments, the server 804 includes a 
transcoder 808 which is configured to down-sample images from a size M x 
N DCT image to a size 1x3 DCT image. In some embodiments, the M x N 
DCT images are images received from the Internet. 

[00129] The transcoder 808 comprises a processor 810 and a memory 
812. The processor 810 may be configured to execute the steps of any of 
the preceding methods to find transform matrices for down-sampling a 
DCT image directly in the DCT domain. The processor 810 may also be 
configured to down-sample a DCT image directly in the DCT domain using 
the transform matrices. In one embodiment, the memory 812 is used to 
store transform matrices. The memory 812 may also be used to store any 
of the optimization parameters. 

[00130] The processor 810 is connected to the memory 812. In one 
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embodiment, the processor is configured to select a spatial domain down- 
sampling method and to apply the selected spatial domain down-sampling 
method to a DCT image to produce a DCT reference image. The processor 
810 may be configured to store optimization parameters which have been 
found to minimize or reduce the optimization problem to the memory 812. 

[00131] The processor 810 may be configured to determine or further 
optimize transform matrices during periods when the processor is 
experiencing little use. 

[00132] In some embodiments, the processor 810 is configured to 
produce a down-sampled image of size 1x3 according to the method 
illustrated in Figure 7 and described above. 

[00133] In some embodiments, the transcoder is included on the 
mobile device 802. The mobile device 802 also has a display for displaying 
images and a communication module for communicating with the server. 
The mobile device 802 may also include a processor connected to the 
communication module. The processor is configured to execute the down- 
sampling method illustrated in Figure 7. That is, the processor is 
configured to obtain from the server the transform matrices and to 
determine an intermediary matrix as the product of the DCT image and 
one of either the row transform matrix or the column transform matrix and 
to determine a down-sampled DCT image of size 1x3 as the product of the 
intermediary matrix and the transform matrix which was not used in 
determining the intermediary matrix. This embodiment, in which the 
server determines and provides the transform matrices to the mobile 
device 802 which then performs multiplication operations using the 
transform matrices may be particulariy useful for down-sampling images 
which are locally produced on the mobile device using, for example, a 
camera. This configuration permits a user to down-sample a locally 
produced image obtained using the camera to minimize the bandwidth 
used in transmitting the image over the network. Accordingly, in some 
embodiments, the mobile electronic device 802 Includes a camera. 

[00134] Accordingly, in one aspect the present application provides a 
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method of down-sampling a DCT image of size 1^ x N in the DCT domain to 
produce a down-sampled image of size I x J comprising steps of: a) 
obtaining transform matrices comprising a row transform matrix of size I x 
M and a column transform matrix of size N x 3; b) determining an 
intermediary matrix as the product of the DCT image and one of either the 
row transform matrix or the column transform matrix; and c) determining 
the down-sampled DCT image as the product of the intermediary matrix 
and the transform matrix not used in the previous step of determining. 

[00135] In another aspect, the present application provides a 
transcoder for transcoding a DCT image. The transcoder has a memory 
having transform matrices stored thereon. The transform matrices 
comprise a row transform matrix of size I x 1^ and a column transform 
matrix of size N x J. The transcoder also includes a processor connected to 
the memory and configured to obtain from the memory the transform 
matrices, and to determine an intermediary matrix as the product of the 
DCT image and one of either the row transform matrix or the column 
transform matrix. The processor is also configured to determine a down- 
sampled DCT image of size I x J as the product of the intermediary matrix 
and the transform matrix not used in determining intermediary matrix. 

[00136] In another aspect, the present application provides a mobile 
electronic device having a transcoder for down-sampling a DCT image. 
The mobile electronic device includes a display for displaying images. The 
mobile electronic device includes a communication module for 
communicating with a server and a processor connected to the 
communication module. The processor is configured to obtain from the 
server the transform matrices including a row transform matrix and a 
column transform matrix. The processor is also configured to determine 
an intermediary matrix as the product of the DCT image and one of either 
the row transform matrix or the column transform matrix. The processor 
is also configured to determine a down-sampled DCT image of size I x J as 
the product of the intermediary matrix and the transform matrix not used 
in determining the intermediary matrix. 
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[00137] In yet a further aspect, the present application provides a 
method of selecting transform matrices for use in a transcoder for down- 
sampling a DCT Image directly in the DCT domain. The method comprises: 
a) obtaining a training set by applying a pre-selected down-sampling 
method to a DCT reference image to obtain a down-sampled DCT 
reference image; b) initializing the transform matrices; and c) applying a 
learning with forgetting algorithm to the transform matrices to obtain 
transform matrices which reduce a learning objective function, wherein the 
learning objective function is a function of the visual quality of a down- 
sampled DCT image obtained using the transform matrices relative to the 
visual quality of the down-sampled DCT reference image, and the learning 
objective function is also a function of the computational complexity of 
applying the transform matrices. 

[00138] In yet another aspect, the present application provides an 
electronic device having a memory for storing transform matrices and a 
training set. The transform matrices include a row transform matrix and a 
column transform matrix. The training set includes at least one DCT 
reference image and a corresponding down-sampled DCT reference image 
obtained using a spatial domain down-sampling method. The electronic 
device also includes a processor connected to the memory for obtaining 
transform matrices for down-sampling a DCT image directly in a DCT 
domain. The processor is configured to initialize the transform matrices 
and to apply a learning with forgetting algorithm to the transform matrices 
to obtain transform matrices which reduce a learning objective function. 
The learning objective function is a function of the visual quality of a down- 
sampled DCT image obtained using the transform matrices relative to the 
visual quality of the down-sampled DCT reference image. The learning 
objective function is also a function of the computational complexity 
associated with down-sampling using the transform matrices. 

[00139] In another aspect, the present application provides a method 
of selecting transform matrices for use in a transcoder for down-sampling 
a DCT image of size M x N to a down-sampled DCT image of size I x J 
directly in a DCT domain. The method comprises: a) selecting a spatial 
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domain down-sampling metliod; b) applying the selected spatial domain 
down-sampling method to the OCT image to produce a down-sampled DCT 
reference image; and c) selecting transform matrices that minimize an 
optimization problem, wherein the optimization problem is a function of 
the error between down-sampled image obtained using the transform 
matrices and the down-sampled DCT reference image, and the 
optimization problem is also a function of the computational complexity of 
applying the transform matrices, the transform matrices comprising a row 
transform matrix and a column transform matrix. 

[00140] In another aspect, the present application provides an 
electronic device. The electronic device Includes a memory for storing 
transform matrices. The transform matrices include a row transform 
matrix and a column transform matrix. The electronic device further 
includes a processor connected to the memory for obtaining transform 
matrices for down-sampling a DCT image of size M x N to a down-sampled 
DCT Image of size I x J directly in a DCT domain. The processor is 
configured to select a spatial domain down-sampling method and to apply 
the selected spatial domain down-sampling method to the DCT image to 
produce a DCT reference image. The processor is also configured to store 
transform matrices to the memory which minimize an optimization 
problem. The optimization problem is a function of the error between 
down-sampled image obtained using the transform matrices and the DCT 
reference image. The optimization problem is also a function of the 
computational complexity of applying the transform matrices to down- 
sample the DCT Image. 

[00141] Other aspects and features of the present application will be 
apparent to those of ordinary skill in the art from a review of the following 
above description when considered in conjunction with the drawings. 

[00142] Certain adaptations and modifications of the invention will be 
obvious to those skilled in the art when considered in light of this 
description. Therefore, the above discussed embodiments are considered 
to be illustrative and not restrictive, the scope of the invention being 
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Indicated by the appended claims rather than the foregoing description, 
and all changes which come within the meaning and range of equivalency 
of the claims are therefore intended to be embraced therein. 
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WHATIS CLAIMED IS: 

1. A method of down-sampling a DCT image of size M x N in the DCT 
domain to produce a down-sampled image of size I x 3 comprising: 

obtaining transform matrices comprising a row transform matrix of 
size I X M and a column transform matrix of size isi x J; 

determining an intermediary matrix as the product of the DCT 
image and one of either the row transform matrix or the column transform 
matrix; and 

determining the down-sampled DCT image as the product of the 
intermediary matrix and the transform matrix not used in the previous 
step of determining. 

2. The method of claim 1, wherein the steps of determining transforms 
the DCT image to the down-sampled DCT Image directly in the DCT 
domain. 

3. The method of claim 1 wherein the DCT image Is a JPEG image. 

4. The method of claim 1, wherein the DCT image is a frame of a 
video. 

5. The method of claim 4, wherein the video is of the DV video type. 

6. The method of claim 1, wherein the row transform matrix and the 
column transform matrix are matrices that have been selected by solving 
an optimization problem which jointly optimizes the visual quality of down- 
sampled DCT images obtained using the matrices and the computational 
complexity associated with applying the matrices to the DCT images, and 
wherein the optimization problem includes at least one parameter relating 
to visual quality and at least one parameter relating to computational 
complexity. 

7. The method of claim 6, wherein the at least one parameter relating 
to visual quality of one of the down-sampled DCT images includes a 
measurement of difference between the one of the down-sampled DCT 
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images and a reference down-sampled image obtained using another 
down-sampling method. 

8. The method of claim 1, wherein obtaining the transform matrices 
comprises applying a decay to elements of the transform matrices to 
obtain a desired balance between visual quality and computational 
complexity, the desired balance being specified by a quality and complexity 
trade-off parameter. 

9. The method of claim 1 wherein the row size matrix and the column 
size matrix have been determined by using a multi-layer network to solve 
an optimization problem. 

10. The method of claim 1, wherein the step of obtaining includes 
looking up the row transform matrix and column transform matrix in a 
memory. 

11. The method of claim 1, wherein the step of obtaining includes 
looking up appropriate transform matrices in a database of transform 
matrices. 

12. The method of claim 1, wherein the step of obtaining includes 
requesting appropriate transform matrices from a server and receiving the 
row transform matrix and column transform matrix from the server. 

13. The method of claim 12, wherein the step of requesting appropriate 
transform matrices from the server includes transmitting system 
identification data to the server. 

14. The method of claim 13 wherein the identification data includes data 
identifying a display resolution. 

15. The method of claim 13 wherein the identification data includes data 
identifying a processor speed. 

16. A transcoder for transcoding a DCT image comprising: 

a memory having transform matrices stored thereon, the transform 
matrices comprising a row transform matrix of size I x M and a column 
transform matrix of size N x J; and 
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a processor connected to the memory and configured to obtain from 
the memory the transform matrices, and to determine an Intermediary 
matrix as the product of the DCT Image and one of either the row 
transform matrix or the column transform matrix, and to determine a 
down-sampled DCT Image of size I x J as the product of the intermediary 
matrix and the transform matrix not used In determining intermediary 
matrix. 

17. The transcoder of claim 16, wherein the row transform matrix and 
the column transform matrix are matrices that have been selected by 
solving an optimization problem which jointly optimizes the visual quality 
of down-sampled DCT images obtained using the matrices and the 
computational complexity associated with applying the matrices to the DCT 
images, and wherein the optimization problem includes at least one 
parameter relating to visual quality and at least one parameter relating to 
computational complexity. 

18. A mobile electronic device having a transcoder for down-sampling a 
DCT image comprising: 

a display for displaying images; 

a communication module for communicating with a server; and 

a processor connected to the communication module and configured 
to obtain from the server the transform matrices including a row transform 
matrix and a column transform matrix, and to determine an intermediary 
matrix as the product of the DCT image and one of either the row 
transform matrix or the column transform matrix, and to determine a 
down-sampled DCT Image of size I x 3 as the product of the intermediary 
matrix and the transform matrix not used in determining the intermediary 
matrix. 

19. The mobile electronic device of claim 18 further comprising; 

a memory having device Identification information stored thereon, 
and wherein the processor is configured to transmit the identification 
information to the server to obtain the transform matrices. 
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20. The mobile electronic device of claim 18, wherein the row transform 
matrix and the column transform matrix are matrices that have been 
selected by solving an optimization problem which jointly optimizes the 
visual quality of down-sampled DCT images obtained using the matrices 
and the computational complexity associated with applying the matrices to 
the DCT images, and wherein the optimization problem includes at least 
one parameter relating to visual quality and at least one parameter 
relating to computational complexity. 
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AMENDED CLAIMS 
received by the International Bureau on 10 October 2008 (10.10.2008) 

WHAT IS CLAIMED IS: 

1. A method of down-sampling a DCT image of size M x N In the OCT 
domain to produce a down-sampled OCT image of size 1x3 comprising: 

obtaining transform matrices comprising a row transform matrix of 
$iz& I X M and a column transform matrix of size N x J; 

determining an intermediary matrix as the product of the DCT 
Image and one of either the row transform matrix or the column transform 
matrix; and 

determining the down-sampled DCT image as the product of the 
intermediary matrix and the transform matrix not used In the previous 
step of determining. 

2. The method of claim 1, wherein the steps of determining transforms 
the DCT image to the down-sampled DCT image directly in the DCT 
domain. 

3. The method of claim 1 wherein the dct image is a 3PEG image. 

4. The method of claim 1, wherein the DCT image is a frame of a 
video. 

5. The method of claim 4, wherein the video is of the DV video type. 

6. The method of claim 1, wherein the row transform matrix and the 
column transform matrix are matrices that have been selected by solving 
an optimization problem which jointly optimizes the visual quality of down- 
sampled DCT Images obtained using the matrices and the computational 
complexity associated with applying the matrices to the DCT images, and 
wherein the optimization problem includes at least one parameter relating 
to visual quality and at least one parameter relating to computational 
complexity. 

7. The method of claim 6, wherein the at least one parameter relating 
to visual quality of one of the down-sampled DCT images includes a 
measurement of difference between the one of the down-sampled DCT 
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images and a reference down-sampled Image obtained using another 
down-sampling metfiod. 

8. The method of claim 1, wherein obtaining the transform matrices 
comprises applying a decay to elements of the transform matrices to 
obtain a desired balance between visual quality and computational 
complexity, the desired balance being specified by a quality and complexity 
trade-off parameter. 

9. The method of claim 1 wherein the row size matrix and the column 
size matrix have been determined by using a multi-layer network to solve 
an optimization problem. 

10. The method of claim 1, wherein the step of obtaining includes 
looking up the row transform matrix and column transform matrix in a 
memory. 

11. The method of claim wherein the step of obtaining includes 
looking up appropriate transform matrices in a database of transform 
matrices. 

12. The method of claim 1, wherein the step of obtaining includes 
requesting appropriate transform matrices from a server and receiving the 
row transform matrix and column transform matrix from the server. 

13. The method of claim 12, wherein the step of requesting appropriate 
transform matrices from the server includes transmitting system 
identification data to the server. 

14. The method of claim 13 wherein the identification data Includes data 
identifying a display resolution. 

15. The method of claim 13 wherein the identification data includes data 
Identifying a processor speed. 

16. A transcoder for transcoding a OCT image of size M x N to produce a 
down-sampled OCT image of size I x ] comprising: 

a memory having transform matrices stored thereon, the transform 
matrices comprising a row transform matrix of size I x M and a column 
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transform matrix of size N x J; and 

a processor connected to the memory and configured to obtain from 
the memory the transform matriceSy and to determine an intermediary 
matrix as the product of tlie DCT image and one of either the row 
transform matrix or the column transform matrix, and to determine the 
down-sampled DCT Image of size I x J as the product of the intermediary 
matrix and the transform matrix not used In determining intermediary 
matrix* 

17. The tran$coder of claim 16, wherein the row transform matrix and 
the column transform matrix are matrices that have been selected by 
solving an optimization problem which jointly optimizes the visual quality 
of down-sampled DCT Images obtained using the matrices and the 
computational complexity associated with applying the matrices to the DCT 
images, and wherein the optimization problem includes at least one 
parameter relating to visual quality and at least one parameter relating to 
computational complexity. 

18. A mobile electronic device having a transcoder for down-sampling a 
DOT Image comprising: 

a display for displaying images; 

a communication module for communicating with a server; and 

a processor connected to the communication module and configured 
to obtain from the server the transform matrices including a row transform 
matrix and a column transform matrix^ and to determine an intermediary 
matrix as the product of the DCT image and one of either the row 
transform matrix or the column transform matrix, and to determine a 
down-sampled DCT image of size I x 3 as the product of the intermediary 
matrix and the transform matrix not used in determining the intermediary 
matrix. 

19. The mobile electronic device of claim 18 further comprising: 

a memory having device Identification information stored thereon, 
and wherein the processor is configured to transmit the identification 
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Information to the server to obtain the transform matrices. 

20. The mobile electronic device of claim 18, wherein the row transform 
matrix and the column transform matrix are matrices, that have been 
selected by solving an optimization problem which jointly optimizes the 
visual quality of down-sampled DCT images obtained using the matrices 
and the computational complexity associated with applying the matrices to 
the DCT images, and wherein the optimization problem Includes at least 
one parameter relating to visual quality and at least one parameter 
relating to computational complexity. 
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